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ABSTRACT 

This report concerns the governance of the structure 
of the National Assessment of Educational Progress (NAEP) , especially 
the law and the role of- the Assessment Policy Committee (APC) . The 
major sources of evidence include interviews with APC members and 
National Institute of Education (NIE) staff, minutes of APC meetings 
and historical documents on NAEP, statutes, and field research on use 
of NAEP. Several options for change are, suggested that involve 
minimal disruptions of current operations: (»J the categories Of APC 
membership specified by law should include an expert in statistical 
research policy;, (b) the APC should have direct, routine access to a 
senior NIE executive who is responsible for liaison with NAEP and for 
articulating and explaining NIE's agenda; (c) a formal mechanism for 
assuring that APC members are aware of new issues and i^^^s, apart 
from those provided by NAEP and NIE, needs to be created; and (d) the 
tension between NIE's research mission and the NAEP's traditional 
data collection mission needs to be resolved. Several 
major change include those that focus more heavily on 
(b) cooperative statistical systems and research, and 
(Author/PN) 



options for a 
(a) research, 
(c) states. 
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1. Introductign 



The National Assessment of Educational Progress (NAEP) was designed 
primarily to "detect current status and report changes in the educational 
attainments of young Americans." Over the past 13 years, the public, 
through the federal government, has invested overi^O million in this large- 
scale program of achievement testing program and in the- distribution of 
results. 

Under current ^law, federal funds for NAEP are awarded through competi- 
tive contracts or other funding mechanisms through the National Institute of 
Education, The Education Commission of the States, which has run NAEP 
since the survey began, received its most recent? three year contract in 
1979. NAEP operations are managed by a specialized group, hereafter desig- 
nated NAEP management and staff, that is a part of the Education Commission 
of the States. Actual survey operations and technical work are subcontracted 
to other organizations such a& the Research ' Triangle Institute. 

1.1 The Focus of this Report ' 

This report concerns the governance structure of the National Assessment 
especially the law and the role of the Assessment Policy Committee (APC) . 
In particular, we address the following question: 

What alternative governance structures .. .will allow 
(a) NAEP to deal analytically and evaluatively with 
contemporary educational issues and (b) minimize the 
changes that NAEP can .become a federal tool leading 
to develbpment of federal standards, curricula, and tests? 
The qu'^stion was framed by the National Institute of Education based 
on their responsibility to understand how NAEP can be operated better, 
' whether any governance options would indeed improve operations or products. 
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vhether any governance options would indeed improve operations or products, 
and the senses in which they might be better. The qualification on the 
question, concerning federal involvement ,, is important in some respects 
and trivial in others. It is also discussed in the following remarks. 
An immediate justification for posing the question is a review of NAEP 

Q 

by Wirtz and LaPointe (1982) that implied that the' queit ion is an important 



one , 



This report" relies on interviews with APC members , NIE staff, congres- 
sional staff and "others, on the contents of documents such as minutes of 
the APC and. the federal statute authorizing . the Assessment, and on our own 
^ earlier research on the use of NAEP products (Sebring & Boruch, 1982). , 



'1.2 Elements of the Curren"r Governance S truet-ure 



Federal law*'"(26 USC 1242) detlrmines the governance structure of 
NAEP in a fundamental way. In brief, the Congress: 

(a) makes the National Institute of Education- responsible for 
carrying out the survey, 

(b) provides that the survey be carried out through an education 
organization to whom funds are awarded for the purpose by 
NIE, 

(c) provides that the education organization create an Assessment 
Policy Committee to govern its own operations. 

The law itself and these three elements constitute the crude governance 
structure. It is peculiar, for the reasons discussed below. 
The Institute Role 

The NIE is' required by law to "carry out" NAEP. As a matter of 
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practice, the phrase implies identifying and' selecting a grantee for the 
work, assuring that the award process is a fair one, and assuring that 
the grantee is accountable in some sense. 

The onlx. further legislation specification of NIE ' s role is given in 
the section of law dealing with support of NAEP. The monies for conducting 
NAEP can be disbursed by NIE through "grant or cooperative agreement with 
a nonprofit education organization." The exclusion of the option to award 
contracts rather than grants can be Interpreted reasonably as congressional 
interest in avoiding unnec^essary federal influence-on a national assessment. 

Congress, shifted control of NAEP .'.monies from NCES to NIE in 1979. 
.Before 1979. awards were made on t^e. basis of cooperative agreements between 
NCES and the Education Commission of « the States. That is, an essentially 
sole source grant has been awardedVoutinely . In 1979, NIE initiated 
efforts to put the process into a competitive grant framework more compati- 
ble with the practices of a research-oriented Institute and the ^eer review 
process that research awards engender. The single bidder at that time was 
ECS and it was awarded support for operations between 1980-83. 

One reason for the shift in control from NCES 'to NIE appears to be 
specified partly in Senate Report 95-856. "The Committee (on Human Resources) 
has received numerous expressions of concerns from chief state school 
officers, state board members ... that the National Center for Educational 
Statistics (has) shown a disregard and insensitivity to the fears of federal, 
control of the programs...." There are other reasons for the shift; they - 
are not specified in the Report. 
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The Education Organization 

The law specifies very little about the education organization to 
which awards are .made. The. only requirements are that it: 
(a) be nonprofit ' , 

" (b) "be responsible for overall management of the National 

Assessment . • 

(c) "delegate authority "elsewhere to design and supervise the conduct 

of NAEP.-. 

The Education Commission of the States has been the education organization 
since the inception of NAEP. A staff, directed by Roy Forbes, has taken 
responsibility for NAEP within ECS. The. staff is aided considerably by 
consultants. 

Th e Assessment Policy Committee , 

The law requires that°the education organization responsible for 
overall management of NAEP delegate authority to an Assessment Policy 
Committee to "design and supervise the conduct of the National Assessment." 
The composition of the AFC is also prescribed by law to include state 
legislators', classroom teachers, and other categories discussed below. 

The arrangement is peculiar in that APC authority is indirect: 
the manager of the study delegates authority to a committee in whose 
selection the manager may play a role. 

It is peculiar as well in that responsibility for creating the APC 
is not given to any federal agency. Again, the reason appears to lie 
in Senate Report on NAEP... "to make unmistakably clear the Committee's 
intent that the progrkm is to be carried out with extensive involvement 
of the education community at the state and local levels." 



1.3 Premises ' 

This report is based on three major premises. 

First, we believe changes in NAEP's governance structure are war- 
ranted only if such changes will improve the product, its usefulness, 
or the efficiency of its production. To put it .bluntly, if the wheel 
is not broken, we should not attempt to fix it. 

The basic NAEB products are statistical data on academic achievement , 
and the distribution of statistical results. Secondary products, not 
expected early in NAEP ' s development but important nonetheless, ap pear 
to be considerable technical and nontechnical assistance , to state and often 
local education agencies witb responsibility for monitoring, educational 
progress. Other products are possible and can enhance" the utility of the 
bas;Lc product remarkably. But whether these can or should be produced is 
debatable. 

The second premise is that the Congress' shift of control of NAEP 
monies from NCES to NIE can be regarded as an increased stress on a research 
orientation rather than just data collection. NIE is, after all, the . 
federal agency responsible for basic research in education and a fair 
amount of applied work that is eventually exploited at local, state, and 
federal levels. In part, the research perspective is reflected in this 
report's topic. There were in fact other reasons, concerning governance, 
for the shift in control from NCES to NIE. They are considered in the 
following remarks only in the context „of relations between the federal 

c 

government and the NAEP operation. 

The' third premise is a recognition that NAEP has developed, over the 
past 15 years, from an idea to a stable and informative device for monitoring 
achievemen: . It is remarkable in this respect, and the evidence on its 



usLe, if not utility, is ample. Still, recent reports and-criticism 
of NAEP provide an opportunity for review and perhaps innovation. 



9 



' ' 2. The Law and Structtrre 

the law'itself is a source of tension as well as a source of 

opportunity in NAEP governance. There are discrepancies as well as con- 

cordance' betweea what the law prescribes and what actually happens. 

■ . . ■ . k: 

2.1 The Law as a Fiindamental Source of Tension and of Opportunity . 

The letter of the law specifies that \ 

(a) NIE "shall carry out" NAEP; ' ; 

(b.) the.educati6n organization through which this is done^^^^shall 

be responsible for overall management af the National Assessment;" 
. (c) the "Assessment Policy Committee (will) design and "su^^^rvise 
the conduct of the' National Assessment." 
The language of the law does not help one to understand where the limits 
of authority lie. Its lack of clarity is likely to be one cause of tensions 
among federal agencies responsible for NAEP and the education organization 
responsible for management cind^the APC responsible for design and supervi- 
sion. 

At Worst At its worst, the law can be regarded as creating a Eiefdom with 
three masters, each jealously guarding their powers. It is an untidy • 
arrangement to judge" from squabbles between NAEP and the federal government. 
But the constitutional form of legitimated mistrust has worked in other . 
quarters for two hundred years. The form invented for NAEP does work in 
limited Ways too. ' ■ 

At Best At its best, the dispersal of ^thority can. be regarded as an oppor- 
tunity for better .collaboration among individuals with .ari interest in American 
education. To judge from the. comments of Assessment Policy Committee members, 
the system works In this respect too, at least up to a point. 
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2.2 Responsibility versus Capability of the Assesstne^t'Policy Committee 

The APC is required by law to be responsible for tasks with which 
members cannot possibly deal directly . - 

To judge from the minu.tes of APC meetings .and "from interviews the 
topics that-^receive most attention ftom the^ APC ^d that are compatible 
with the law's requirements involve choice of substantive ar'eas for ■ ' 
testing (including target sample characteristics^ such as age group) and 
the NAEP budget. The serious attention given to these areas is clear. 

There are excursions into other topical areas but these are much 
less intense and less uniform and predictable. Many ACP meetings, for 
example, involve 'merely providing information by NAEP staff rather than . 
formal policy, decisions , e.g. discussion of press coverage of a particular - 

survey, * 

other areas that are denominated in the law receive far less attehtion" 
than testing areas and budget. More important, these areas are very complex. 
Making the APC "responsible" for them in the law seems gratuitous at best. 

For example, the law says that the APC is "responsible" fior the 
development and selection of goal statements and assessment items, the 
assassment methodology, the form and the content of the reporting and 
the dissemination of assessment results, and studies, to evaluate and improve 
the form and utilization of NAEP" (Paragraphr D3 Section 405). It is diffi- 
cult to see how responsibility for these tasks can be taken, other than 
superficially, simply because the APC is not composed of experts in these 
areas. f More important, Selection of goa,l statement^, test items and ■ 
the like are simply not "policy" issues, as the phrase is generally used. 
For example, no members of APCs have played 'a sufficiently major role 
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■ . \ 

'in designing utilization studies or evailuatlons of SAEP to be acknowledge* 

' in published report^ of those studies. At best, they have provided reac-. 

tions to NAEP staff suggestions, and perhaps -^ven infl^ienced a choice to 

pVsue one o^ -several^ op tions . No"" members " of APC appear to have done 

research on dissemination or on goal statements and test items that is 
S' .< r '.' 

" suffici^entry intens'ive' to be reported in schcTlarly journals. Again, at best 

• ' * » * 

they" may^ provide reactions to NAEP management and staff. ^ ' ; , 

The problem of discrepancy of course does not lie with the APC. . . 
Part' of the. problem liW in law 'that asks individuals to take responsibility^ 
for matters' that lie beyond their ken, matters for which even if individuals 
did have the pertinent" expertise, tlm'e alone would not permit serious res- 
ponsibility. Paft of. the problem of course also lies in the ambiguity of 
the word ^'responsr^ility" as it-'s used in the law. 



r 

Options 



\ 



The options for^ change here* include; 

i; Defining -responsi-bility so as to mean only oversight 
guided by questions that are properly policy. 

2. Narrowing responsibility to focus attention only in 
broad area-s with which the APC -is best able to deal, 

^ notably learning areas; budget and, more important, broad 
pol?.cy he'aring on better exploitation of NAEP,, 

3. Changing th& APC ^composition, budget structure, and 
operations to match between responsibility and capacity. 

2.3 The NAEP - Feder.al Linkage • . ■ - 

\ ■ ' \ . 

The Mational 'institute of Education is responsible, under the law, for 

, ■ ■ • 

carry'irig out "by g;:ant to or cooperative agreeipent . . .with a nonprofit. 'educa- 
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tion organization, a. National Assessment of Educational Progress .. .The 
education organization through which the Institute carries out the Assess- 
ment shall be responsible for overall management of the Assessment" (Sec. 
1242, Section 405 of the General Education Provisions Act"). This prose 
leaves a fair amount of leeway for relations between NIE and other parts 
of the governance structure - NAEP managemeiit and staff;Jn'll the APC. 
-'. There is a source of tension in those relVions that doe§ not seem 
necessary or productive. It turns around the view of at least somep?C 
members'that NIE policy on NAEP is not clear. Specific questions that 
members posed during my conversations included: , 

. What is NIE's agenJa for NAEP and how is it to be constructed? 
Why has NIE chosen to open the award process to competitive 
bidding? 

. Why has NIE emphasized production of - - 

» O 

, Public use data tap^s 
, Issue papers 

when budgets have not been adequate, APC members maintain, to produce the 
data- that is needed. Part of the tension appears to stem from the fact 
that the APC is not informed about the rationale for NIE action or the 

o 

sources of pressure and responsibility that imply action must be taken. 

For instance, the law requires that NIE's Director be an ex-officio 
member of the Assessment Policy Committee. This is the only fomal channel 
for communicating federal interests to the APC and vice versa. It is an 
inadequate one simply because the turnover in NIE Directors is high: most 
last less tharv two years in their position. The same problem affects the 

m 

V development of coherent NIE policy on policy on NAEP. 
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Explanations that address the more specific questions have indeed been 
addressed by NIE staff, to judge from minutes of APC meetings. Still, it 
appears that better explanation would help. The fact that public use data 
tapes were promised by NAEP over five years ago, but were not delivered 
is important and does not seem to have been caken into account in criticism 

of the Institute. 

A second more generic source of tens^ion is that NIE is a research 
support Institute. Unlike NCES, it is not an agency for routine collection 
of data. The research orientation implies that NIE staff will 'be strongly 
interested as they should be in new ideas, bearing on the use of ^AEP for 
instance, the different directions NAEP could take, and so on. The respon- 
sibility for NAEP is then misplaced unless either NAEP- becomes more research 
oriented or the mission of NIE is broadened. The latter seems silly in 
-that NCES has had -responsibility for^general education s urveys. 

c 

3. APC Operations 

The time available for the APC to consider issues has been considered 
by 'some critics to be inadequate. The APC's normal access to information 
seems at least as important. Both are considered here briefly, since they 
are influenced by law. 

3.1 Time and APC Responsibility 

APC members may, under current law, serve for three years and can 
serve for no more than two consecutive terms. Members are asked to meet 
.three times each year under current by-laws, with meetings typically being 
a day and a half or two days duration. 

To judge from interviews with APC members, they regard their terms and 

14 
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time available as adequate for the kinds of issues they are asked to 
address, but some would prefer more time. In particular, the time is 
viewed as adequate for broad policy choices. More time is warranted to 
the extent that the Committee is asked to or prefers to dig deeper into 
issues. 

It is not clear that changes In structure of governance should be 

c. 

made to affect duration of terms, frequency of meetings, and the like. 
In particular, the APC itself seems the most appropriate body to lay out 
frequency and duration of meetings. Outside mandates are likely to be 
gratuitous . In Wy cas e , f requency ffi are iriarter s~X)f 'AP C""" — 

by-laws rather than statute or federal regulation. 

Proper term of office is as debatable here as in other cases of appoint- 
ments to government advisory committees. It is a matter of law and therefore 
changeable. No remarkable reasons for change, however, surfa ced during this 
study- 

Some changes that could be justJ.fied are applicable only if the product 
or its production are deemed unsatisfactory in a few respects. For example, 
if the language of the law is subscribed to vigorously, then longer terms for 
some members may be warrarvj^^d^ the interest of stability; shorter terms 
for others warranted inVrlie interest of freshness. Changing by-laws, however 
to meet this contingency s^fernBYnore practical than changing law. 

3.2 Augmenting APC Capacity 

The capacity of the APJC to deal with issues is influenced partly by - its 
. willingness and capacity to seek assistance. The APC has gen erally relied 
on (a) NAEP staff, (b) Technical Advisory Boards, and (c ) periodically on 
outsiders . . 

15 
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The APC relies heavily for its information on NAEP management and 
sta£f , to judge from minutes of APC meetings and interviews with APC 
members. However, NAEP management and staff have been willing at times to 
assure that the APC is presented with reports from outside the NAEP struc- 
ture. More important, NAEP management and staff have provided the APC with 
access to information that is not always complimentary. NAEP Director Roy 
Forbes, for example, invited Archie LaPointe to report preliminary results 
-of the Wirtz-LaPointe review of NAEP to full committee in 1981. 

None of the APC members to whom we spoke were dissatisfied with the 

■"TirfofS^i5^^pr5^idia--W^ 
On the contrary, there was a fair amount of praise for each. This does not 
mean the APC has always been in accord with opinions of NAEP management-and-^ 
staff - there have been disagreements. But the endorsement is strong. 

De spite this , there is no real guarantee that fundamental criticism 



will surface and receive a fair hearing from the APC, that fundamental ques- 
tions about NAEP will be addressed, or that options other than those thought 
suitable by NAEP management and staff will be considered or recognized. 
Hearings before House Committee on Education in 1977 make it plain that the 
APC is "expected" to seek out advice of experts in educational research, 
higher education and statistics, presumably in these matters and others. ^ 
The underexploited source qf • information involves outside expertise. 
* During Interviews with APC members, some acknowledged that indeed they do 
not know the area of educational statistics well enough to know where to 
seek independent counsel. Others maintained that the organizations they 
represent were the only source of independent counsel they need. (In inde- 
pendently checking one of these organizations, it became clear. to this 
writer at least that no member had contributed remarkably to educational 
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statistics or research policy or methods,) Money that NIE had allocated 
to independent consultants for the APC has gone unused; indeed one member 
failed entirely to remember it during a telephone interview. 

The need to augment APC capacity, through law or other mechanisms, 
then is ambiguous. APC members generally praise NAEP and NAEP conferences 
as an information source. And for routine data collection these sources 
are likely to be sufficient. It is arguable then that augmentat j.on is 
unnecessary. 

The arguments for augmentation are that predicated on the beliefs 
-th-atr(-i^-the-AFG-s-hould consider new ideas for NAEP and (2) new ideas stem 
from outside the NAEP management staff, conferences, and support system. 
NAEP does exploit a great- many consultants responsible for production of 
technical ideas. Still, it is a legitimate argument, and evidence that 
new ideas do not have suffici^-nt opportunity to emerge are given later. 



3.3 Assessment Policy-.Committee Size and Composition 
Current law requires that the APC be composed of 
, two representatives of business and industry 
. two representatives of the general public 

one chief state school officer 
. two state legislators 
. two school superintender.ts 

. one chairman of a state, board of education 
. one chairman of a local school board 
. one governor of a state 
. four classroom teachers 
The director of the NIE serves as an ex-officio member. 
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Size • 

Though a 17 member committee appears to some observers to be large, 
therefore unwieldy or cumbersome, . the APC members to whom we spoke seem 
satisfied with the arrangement. They maintain they do indeed work well 
together. 

The size of the committee is inflated, of course, in the sense that 
it is rare to have all members present at a meeting. Further, the fact ^ 
that governors (say) do not appear or send representatives may actually 

■ . - ' -\ 

facilitate meetings. ^ 

I see no real grounds for changing number of APC members unless the APC's 
mission is altered greatly. Indeed, for this case as in many others in 
government or commercial enterprise, hard evidence on effectiveness of 
performance of committees, as a function of number of mWers , is non- 
existent. A greater or less number may improve the process or the product -- 
but no systematic~a"dministrative experimentation has been undertaken. 

Composition |j 

Composition of the Committee is a bit more interesting because the APC 
serves some unofficial missions as well as the ones required by statute. In 
particular, it is a symbol^ a signal that local and state influence on NAEP 

must be strong. It is cons iderably more -than symbol, in that APC members 

lobby vigorously, albeit of ten i arochially, for those interests. 

Within the ambit of law and symbol, this still leaves considerable room • 
for improvement . 

The categories listed in the law for instance virtually guarantee that 
very few, if any, will be knowledgeable about statistics, statistical manage- 
ment statistical policy or educational research planning and policy. It can 
be regarded as distressing and absurd that despite 80 years of development 

18 . . 
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in survey sampling, 50 years of technical developments in experimentation, 
and 200 years in policy if we use the Census Bureau as a standard, the cate- . 
.gories specified in the APC ignore them. This is frightening ^hen we recog- 
nize that some $4 million per year are spent on a basically statistical 
enterprise that must be independent and related to of.hers. 

To push the point a bit further, consider that in interviewing APC . 
members we encountered individuals who 

were unaware of the Condition of Education , a massive annual 
report that maps the statistical terrain, exploits NAEP; 

." were unawiFe' of ^hT"tSteht aevelopmgnts ih-Teae^ 

policy bearing on education or society more generally; 
were unaware of NIE's research mission and orientation, as 
distinct from NCES's data collection mission; 

__were- unawana oi_work_Qn_s.ecQndary_analy-sis--that-jus.tdfy 

production of public use data tapes; 
:• were unaware „of the gratuitousness of uses of NAEP, in supporting 
arguments that Title I Compensatory Education programs work. 
None of this should be surprising perhaps. It is alarming in that it suggests 
. no better forms of independence from federal influence will be 

outlined and examined seriously; • , i_ 

.no better arrangements to avoid redundancy with existing systems, 
to learn how NAEP interlocks with existing systems, etc. will be 
developed; 

. no informed skepticism about uses of NAEP in making inferences 
or informed advocacy will prevail; 
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no better ways of reconciling conservative approach that is 
essential to producing educational .indicators with the possibility 
of small Innovations that exploit NAEP better 
Technical advisory group and technical advice does help to avoid some 
problems, if we judge from APG minutes. However, there is no reason to 
expect that individuals who are technically creative to be aware of major 
policy Initiations bearing on statistics and educational Iresearch or to be 

e ■ \ I 

interested in them. . , 

Providing for a research policy specialist on the A?C is no substitute 



for enlargii^g NAEP capabilities for research of course. That requires con- 
siderable funding and a change in NAEP's mission. However, it is likely to 
resuiria. better exploitation of periodic opportunities to enlarge the value 
of NAEP and other woT-lc_that is coupled to it. . 



3. 4 _ 4ihe- Questions -T-hat-^the AP-G- Does and Does-iqot -Ad4;re-ss 

The issues and choices presented for APG deci sions are most frequently^ 
those raised by NAEP management and staff. Mos t are clearly important.. The 
range of neglected issues is great . . 

To judge from APG minutes and interviews, the issues discussed at policy 

meetings are almost invariably those introduced by NAEP__st^ The most _ 

regular topics include budget and choice of areas to be assessed. The ir- 
.regular ories, deciding whether 9-year olds should be tested for example, are 
diverse. 'There is a great deal of reporting, e.g. on press coverage of a 
particular survey.. 



er|c ?o 



two 



offends 



What features of this coverage invite criticism: 

Federal Influence 

,To judge from the minutes, we can rely on one APC member ever}^ 
years to announce rhetorically that-NAEP will not be pushed around jby the 
federal government. The spirit is proper we admit, if we turn to |he law; 
it is specific .in stressing state and local representation on the |PC. The 
lack of arr^ serious intellectual attention to the matter, however,, 
common sense. In particular, never to our knowledge 

(a) examined the extent to which alternative, systematic 
— - ehecks will keep federal influence tolerable, 

(b) examined what "tolerable influence" is or should be, : 

(c) examined policy that would improve cooperation between | 
'federal and NAEP efforts without enhancing federal influpce. 

(d) policy that would better integrate and adjudicate the \ 
■ interests of local, state, and federal ' goverrimint¥, for | 

monitoring children's achievement, and especTaD-y to \ ^ 
recognize incompetence and performance of some groups at 
each of these levels of government. ! 

Evaluation of Schools and ProRrams . From time to time, the use of NAEP 
to evaluate programs is discussed by NAEP. The discussion is onjy 



o7ca7ionally illuminated by people who do know something about the dif- 
ficulty or impossibility of this based on passive data generated; by NAEP. 
For instance, it was announced pro.udly at one APC meeting tjiat NAEP 
~ da^had been used in judging the worth of federally supported Title I 

^rogral^^^No questions arose about the legitimacy of the use, or the mis- 
leading chai^r of the use. There was no recognition of any major 
^ criticism of suchNi^. On the other hand , Ralph Tyler has been .emphatic 
in^taking the positioV^ one meeting, probably more , that NAEP cannot be 
used to judge the worth of JferograjiLS^ - - — ■ — ; 



Similarly, an APC member at a recent meeting announced that the bottom 
33% pf achievement "represents a massive failure of-...the-.school system." No 
one seemed to recognize that (a) somebody will always be in the lower third 
of a distribution, (.b) the statistic does not imply failure of a school 
system anymore than it implies failure of parents or the U.S. Government or 
society, (c) that the standards of evidence in evaluation of schools are not 
met by NAEP data, ard so on. 

1 

Similarly, NAEP management reported on special studies to link achieve- 
ment level data to state expenditures for education. Again, there was no 
recognition that the product of such effort does not permit one to make 
statements about how achievement is improved with expenditures. There was 
no recognition that the simple-minded strategy fosters the naive belief that 
throwing money at pro1)lems resolves them, an embarrassingly common approach 
in federal, state, and some commercial governance. 
"'suidV^wTi^'PiF^iAsmT—^^ a variety of ways, to 



judge from our studies, at the local and state level (Sebring and Boruch, 
1982). It is underexploited in one major respect that has not been con- 
sidered in any depth by the APC. .The exploitation is a policy matter. 

In particular, special studies that are adjoined or capitalized on 
NAEP should be a formal part Tf NAe7 policy at the-federal-,- -s.tate,_ and ^1 
levels, to inform debate over contemporary issues that cannot be addressed 
by NAEP alone. . A mechanism for screening, supporting, and fostering col- 
laboration in such studies should he developed. 

"Deuplication of NAEP for. all or a major part of students within a state 
rather than the usjaal small samples, illustrates the special study on a larg. 
scale. More small scale specialized efforts could be mounted, e.g., investi 
gations of how NAEP test itams match what teachers are trying to teach, or 
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academic research on science achievement based on NAEP data tapes. Such 
arrangements in principle augment NA.^P's usefulness by permitting more 
capitalization on the basic s.-^.sple, rasthods, structure, etc. of NAEP. 

However, there is no coherent plan, policy or policy mechanism for 
encouraging special studies that exploit NAEP. Developing such policy 

O 

seems sensible to foster utilization, because issues do emerge period- 
ically and change, and NAEP can sometimes help address them. So, for 
example, there is no mechanism by which NAEP could be augmented by state- 
wide randomized controlled tests of tuition tax credits, although the NAEP 
samples, administrative structure, tests, etc. are a vehicle for such tests. 

There is no research policy mechanism for exploring the technical issues 

that adjoining such a study raises. There is no management policy on 
screening for opportunities, no clear criteria for judging worth of studies 
that might be adjoined, and so on. 

While this is relevant to federal influence, there is no reason why 
an expert APC and the states could not take a leadership ro\e in this 
endeavor. In states where assessment offices have limited resources to do 
so, or in states' (such as Texas) where independence of the LEA is valued, 
consortium arrangements among LEAs may be possible. Special studies might 
be conducted by anyone with the resources to do a good job, e.g. teachers 
or administrators, PTAs and school boards, academic institutions, etc. . 
" Such aTrangements in. principle augment NAEP's usefulness by permitting 

"""« - 

• more capitalization on the basic sample ," methiods, structure, etfc. of NAEP. 

One major justification for the view that policy on studies. is far 
too provincial stems from NAEP's support.. It Is funded by law through the 

• National Institute of Education. The NIE's mission is research. It must 
■ and should exploit research opportunities where they present themselves. 
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4. The Governance Structure 
Discussed by Wirt z and LaPointe ' 

The Wirtz-LaPointe review of NAEP is pertinent to this repol^t on^^ ' 
governance in two respects. First, Wirtz artd LaPointe raise the possibility 
of creating a council that would change the function and -:haracter of the 
APC or supplant- it.. They acknowledge that their discussion does go beyond 
their original, mandate, and it discussion is very tentative. 

Second. APC members .raised the Wirtz-LaPointe report during interviews 
with this writer. Moreover, one thoughtful APC member suggested that the 
.mrtz-LaPointe-obser-vat-ions-on-frhe-APG -eonsti^^ for renewal 

and ought to be regarded as such. 

Wirtz and LaPointe- (1982) undertook no intensive study of the APC 
operations. 'They neither • praised or criticized Ttdiractly , Rather, criticism 
was indirect in that they proposed an autonomous structure, called an 
Educational Assessment Council, that would have a more generalized mission 
than the current APC has. This Council would be- r^^sponsible for policy and 
decision making (p. 49) and: 

(a) consist of 6-8 "distinguished" individuals, 
including some educators 

(b) meTt'Trequeirfely - 15 Jo 25 days each year 
• (c) maintain a staff of 3-4 professionals 

. (d) be "autonomous" and "independent" ■ > 



24 



22 



It would be charged .with : 

(aV'synthesizing and reporting information produced annually 

by "current educational measurement" because they say the 
. public needs help "in interpreting the bewilder iiig data" (p. 51). 
(b) exercising "broad influence over the kinds of specific 
assessments that are made," a task that does not differ 
from current APC responsibility. 
The Wirtz-LaPointe suggestion is .compatible in some respects with 
remarks made earlier in this report. It is immaterial or incompatible in 
other respects. 

(U - ; . 

Compatibility 

The compatibility appears in four respects: . 
First, there does appear to be a need for more independence of an 
"adv^isorT -or-eoatrpl committee, to avoid almost, exclusive reliance on NAEP 
management or any vesteT iffeeTest . and to^ avoid ■ the discontinuity in APC 
oversight that, is Ukely to be engendered by competitive bidding on. contracts 
that may lead to agencies other than ECS conducting N/VEP. ' 
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' Second . there is a need ior additianal expertise on such committees. 

f ■ 

Third, there is a need for more frequent meetings if^ the committees' 

mission is broadened. 

'.Fourth, Wirtz and Associates stress the need for a committee to exer- 
cise broad "influence on i:Jie_kinds of specific assessments ... undertaken" 
(p. 51). - This too seems sensible based on the remarks earlier in this .report. 

This compatibility does not mean that this report sustains the tentative 
recommendation of Wirtz and' LaPointe for an educational Council. It merely 
reiterates some of the concerns implied by Wirtz and LaPointe. 
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Incompatibility ^ - * ^ 

The lack of compatibility between .Wirtz-LaPointe and this report lies in 



one area. 



Wirtz and Associates^ maintain that we need,' to " find ways to summar- 
ize the^cvo .siderable amounts of data (t-hat) emerge each y.ear" (p. 49). In ^ 

fact, " . 

(a) a system for summarization does indeed exist, and is used, _ 

(b) though the' system is flawed, it is remarkable for its quality, 

, catholicity, and coverage. , . .. > • 

This system consists mainly of annual reports issued by the National Center for 
Educational Statistics under the title Condition of Education. It does, with 
a small staff, a major task Wii;tz et al. want their Council to do; present 
reliable data and facilitate interpretation with prose that summarizes its 
meaning. This system has been criticized both well and badly , "though Wirtz- ' 
LaPointe do not seem to^ recognize this (Wohlstetter and Boruch , . 1981) . Both 
'types of criticism are publicly available, and action has been taken. on at ^ 
least some of the latter \y NCES., Wirtz and LaPointe admire "res-poasible" 
reporting (p. 51), and indeed CE is 'supposed to be a balanced ^rei^ort under 
the -law. It is, by an'd large. v,^ • ^ 

ThP. Condition of Education system does not, however, involve the 
"distinguished" contributors that Wirtz and LaPointe prefer. Rather, the 
producers are f^e^al government staffers, statisticians, and educational 
researchers whose workXs known in' expert circles but who are not -par- 
ticularly' visible at the national level. Moreover, the system does not 
discuss, -create, or evaluate policy as a council might do. Rather its 
purpose is to inform, without editorializing by and large, national and ' - 
state educational policy. " 
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5 . Summary 



This report reviews the possible need for structural changes in govern- 
ance of NAEP and the changes that can be regarded as options. Governance 
here includes law and the major administrative components of NAEP, such as 
the Assessment Policy Committee, The report was requested by NIE, partly 
because there have been no inquiries of a similar kind despite the impor- 
tance of NAEP's governance and because an opportunity for review has been 
created by recent studies of the usefulness of NAEP, The .major sources of 
evidence here include interviews with APC members and NIE staff, minutes of 
APC meetings and historical documents on NAEP, statutes, and field research 

on u^e of NAEP, » ' ^ ^ 

If the traditional products of NAEP are regarded as satisfactory, major 
changes in governance are not likely to be productive. The'^ system works 
sufficiently well to produce (a) reliable data on children's achievement, 
(b) occasionally interpretations , \c) de facto technical educanon for the 
states, and consequently we ought to view major change with ^.^npion^, 

The traditional products of NAEP are less satisfying^ if ' one ' s cri^rion 
includes serious consideration of innovation (aside from samg^ing, testing, 
and public' relations) and more productive relations among the federal, govern 
ment, NAEP, and their constituencies. In particular, new opportunities to 
learn how to exploit data better or more cheaply are not pursued as often 
as they can be. The constraints here include budget,' of course. But some 
aspects of governance are important too. 

. • , Some options sugges^37^ implied hy critics of NAEP do not, appear 
necessary. Changing the number of APC members seems unnecessary in thdt - 
APC members are satisfied with the size and ^o major justification apart 
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from budget is evident. Changing the frequency and duration of meetings 
is the province of the APC and NAEP management, not of law or adminis- 
trative' directive of NIE. H 

Several options exist for change that, involve minimal disruption of 
current operations, do not engender any notable increase in federal in- 
fluence, and open the NAEP system a bit to new ideas. 

(a) First, the categories of APC membership specified by law should include 
an expert in statistical research policy; the absence of such a category in 
law is outlandish and fosters parochialism. .No APC members are recognized 
experts in this arena; technicgkl advisory groups are not substitute for 

0 

membership.' . / 

(b) ' Secortd, the APC should have direct, routine access to a senior NIE ex- 
exutive who is responsible for liaison^with NAEP and for- articulating and 
explaining NIE's agenda. The current' ex-officio membership of the NIE di- 
rector is not satisfactory simply * because political appointments are so 
transient . ^ - «< 

(c) Third, some formal mechanism for assuring that APC members are aware 

of newv issues and ideas, apart from those provided- by NAEP^.and NIE needs to 

be created. *" / ^ " : 

(d) Fourth, the tension Ibetween NIE's research mission and^ the NAEP's ^ .' 
traditionally data collection missioi? -needs to be resolved. Developing formal 
policy t.6 exploit NAEP as a vehicle for research, as a target of research, 

as a producer of data that occasionally will be useful for legitimate re- , 

• -a . . j ■ • 

s'^arch may be ,one way^ of removing some tension "and, more important, de- ^ 
veloping'more dooperation among federal", state, and local interests in NAEP. 
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Several options for major change are worth considering, but tfhey are 
predicated on assumptions about what NAEP should be that are dif fpirtC'-'from 
prevailing assumptions! The changes are not likely to be warranted or even^ 
feasible unless the assumptions are studied mo^e intensively than they have 
been. They can be summarized into those-. that focUs more heavily on (a) re- 
search, (b) cooperative statistical systems and research, and (c) states. 

If NAEP is regarded as a legitimate vehicle for research, as well as 
being regarded a monitoring system, then a variety of major changes seem 
worth considering. None can be done quickly, nor should ' they be implemented 
without discussion and pilot testing. 

The language of tt^e law must be changed to authorize and emphasize 
using NAEP aa a periodic vehicle for research on contemporary problems, 
and to include relevant research expertise on the Assessment Policy Com- 
mittee. The federal government's relation t6 ^ET must be altered to permit 
routine, serious consideration of research proposals by recognized experts. 

Some of the arguments for siich a major change in NAEP emphasis, Chough 
not direction, are negative, e.g. to avoid parochialism, stoginess, and lack 
of imagination that characterize many stable statistical systems and to 
reduce overblown or misleading claims about what such passive systems can I 
do.. The positive arguments include the idea that NAEP is a national re- | 
■source that ought to 'be exploited more effectively, -and moreover that | 
adjoining research projects periodically to NAEP is a device for better ex- 
ploitation and utilization. ' f , ,' 
• . The clear arguments against such change are complexity and cost. The ' 
debatable arguments include possible politicizafion of NAEP, politicization 
that can have either negative or positive benefits. 
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If NAEP is regarded as a small but legitimate part of a large frag- 
mented network of independent local state, and federal data collection 
efforts, then major efforts to identify its role are better warranted. It 
iTn^tTle^r that major changes in governance are warranted to db-TTTrs: 
Rather the task might be accomplished well by creating a liaison committee 
with each federal agency responsible for , research and statistics production, 
a committee with responsibility for identifying why and how NAEP fits into 
other data collection efforts and for identifying opportunities to co- 
ordinate and collaborate. 

If NAEP is regarded solely as a monitoring system and the primary ob- 
jective is to market the system's products to a variety of changing con- 
stituencies, then options for structural change include focusing periodically 
on special audience categories. For example, state legislators are in 
principle a major audience for results, but it is far from clear that they 
ari aware of NAEP. Even if they are 'aware, it is unclear that many would 
be [competent to use the data thoughtfully. Some provision for periodic 

mentation of the existing i^C to direct attention to such audiences, to 
iokertheir interest, provide them with technical assistance and so on 
warranted. The creation of other advisojry__ commit te^^^^ the 

that are" legisla«;.io.n-.^riented, adminisrration oriented, etc. is con- 
vablei No new groups seem warranted without technical support however, 
fhe arguments for " alteration include, the" idea .that an augmented APC or 
ajliary-^c-^mmittees can enhance utility of NAEP, The debatable arguments., 
against alteration include the possible politicization of NAEP. 
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